This question already has an answer here:
Checkout subdirectories in Git?
You should use submodules for this purpose.
Submodulesallow foreign repositories to be embedded within a dedicated subdirectory of the source tree, always pointed at a particular commit.
Break your big project to sub projects as you did so far.
Now add each sub project to you main project using :
git submodule add <url>
Once the projected is added tot your repo you have to init and update it.
git submodule init git submodule update