The Nephilim are mentioned just before the Flood account in
Genesis 6:4, which states: The Nephilim were on the earth in those days—and also afterward—when the sons of God went in to the daughters of humans, who bore children to them. These were the heroes that were of old, warriors of renown.
Mankind started with Adam & Eve (humans). Noah and his family were the only ones left to continue the human race ... there is the line of godly people and ungodly people ... the line "split" those of Abel and those from the line of Cain. All of mankind is from one of those lines. Son's of God are those who served Yahweh ... sons's of men are those who served idols. They were mighty men, strong and large in stature,
Genesis 6:4 Lexicon: The Nephilim were on the earth in those days, and also afterward, when the sons of God came in to the daughters of men, and they bore children to them. Those were the mighty men who were of old, men of renown.
Nothing here depicting the Nephilim as "angels". In fact states clearly they were men (mankind), mighty MEN (mankind)
In the days of Noah there were those who worshiped idols and also afterwards.
The point is there were two lineages ... one from Cain and one from Able, both of whom were human ... and the world was populated via those two lines.