To capture a monster, you must first defeat it in combat. If you proved to the monster that you are worthy of it joining your group, then they will request that they join. You can either accept for decline. You can only have three monsters in your party at a time, the rest of your monsters will be sent here.