I hope that millions of transactions per milliseconds is a joke or a
hyperbole. No single computer can handle that much, particularly if a mere
100Gb counts as ridiculous amounts of RAM.
But ignoring the actual number of expected transactions, what is needed
for this type of task is real-time Java. Provided that you want to stick with
Either way you'll probably need a real-time OS first, because your
application isn't going to be the only thing running on the computer and if
the OS decides not to give your application control when it needs it,
there's nothing you can do.
Update: If you only want to capture network traffic,
don't reinvent the wheel, use
libpcap.. AFAIK there's even a Java wrapper