(Shulchan Aruch Yoreh Deah 265, Siman 113 in the footnotes) “Anyone who does not partake of the meal at a brit mila celebration is as if rejected by Heaven.” (The source of this statement is in the Gemara Pesachim 113… There are seven (types of) people rejected by Heaven … and there are those who say that a person who does not partake of a meal at a mitzva celebration, and Rashbam gives the example of the celebratory meal for a bris mila) and “therefore it is not a good practice to invite people to the meal of a bris mila, because they may not come to it and thus we have put them in the category of being ‘rejected by Heaven’.
Rabbi Chaim Naeh writes (Kovetz Oholei Shem, section 7) that it does not refer only to the meal of a bris mila, but also a wedding meal or any other meal celebrating a mitzva. He also writes that it is a mistake not to invite people to the bris itself, for there is no problem involved in inviting people to the bris itself, only regarding the celebratory meal afterwards, as the Gemara states: “who does not partake” and as the Remo in the Shulchan Aruch writes, “who does not eat, but to invite to the bris and not to the meal, there is no problem.
If a person has been invited to the bris and not to the meal afterwards, he is not in the category of being ‘rejected by Heaven,’ and thus the custom is to serve milei detargimei, drinks and cakes, so that someone who doesn’t stay for the meal will fulfill his obligation by having a taste of something.
Rabbi Naeh further writes that there is an opinion that only someone who is present (at the bris meal) and lessens its importance by not participating in it, is considered (to be in the abovementioned category) but if he wasn’t present he has no obligation at all. There are others who disagree and say that if a person was invited to the meal he has an obligation to come to it.
In summary: One can invite people to the bris, but not to the meal, and if a person has a taste of something there, he has fulfilled his obligation and need not partake of the meal.