I'm sure someone will correct me if I'm wrong but, as far as I'm aware, the limitations on the data in Monarch are, the number of records created in the data view and the amount of resource available on the machine running Monarch i.e. memory etc.
I believe Monarch's internal database in V10.5 falls over at about 9 million rows with a small number of columns per row (according to the Monarch help files). This has increased to 16 million rows in V11.
When you connect to the database in Monarch, there's an option to filter source records (as opposed to creating a filter in a model). I don't know whether this is what you've already tried ?
Alternatively, I do know, from the little that I've used ODBC, that some ODBC drivers can be configured to limit the data that is exposed in the ODBC connection. Unfortunately, not every driver is the same so, it may not apply to your particular scenario.
Sorry I can't be of more help.
You are correct the DB size does not matter. I connect to a DB that is currently 167GB via ODBC connection, every day , all day. The only limitation would be record count of the current record set. If You are unable to filter them via Monarch, create a custom view in your DB to access the data you need.