HI fpeters and welcome to the forum.
V5 is going back a while and I don't have ready access to any limit information but I certainly don't recall a limit that would be as low as 3800 records.
You have checked potential issues related to blanks or no matches. I'm not being funny here but no matter how many times you have done that it may be worth getting someone else to check your logic/process. I have lost many hours over the years doing similar checks and missing something obvious every time it took more than 10 minutes
The other thing might be to ensure that the lookup link is not somehow truncating the input - i.e. it is not actually using the same data you are checking. Typically most likely on the longest keys - does the latest file have any new longer keys? Is that possible the way the links work?
Just thinking aloud at the moment ...
In past I had to redefine the database range in excel to exclude any blank rows. More than one blank row was seen as a duplicate. Since it worked previously with more records than I have now, I doubt that it's related to the number of records.
I checked the table for dup's by doing a summary by the two join fields, and there are none.
I solved the problem by trimming blanks and using concatenation in both Excel and Monarch.
Monarch Report Columns
Detail MFR char 5
Detail CAT char 20
Calc MFRCAT =RTRIM(MFR)+RTRIM(CAT)...... JOIN TO EXCEL MFRCAT
MFR text 5
CAT text 20
No more duplicate values error.