Hi Mike,
Instead of "open in Excel" try using the 'CSV' option and when prompted choose "Save" instead of open in Excel. When I do that using a data source with 300,000 rows, I get all of the rows, even with the default setting of 500 rows for list style report, and 20 rows for a scrolling list style report. Of course, if your report is a drill-down report this would only work from the final detail level. If there's no drill-down path, then you've already started at the detail row level. In either case the value of rows-per-page doesn't matter and is ignored. If the report has a drill-down path, then you'd likely see a message at the bottom of the report grid: this is not the full list. For a drill-down report this indicates bad-practice. Who can navigate 50,000 rows of data in a single browser page anyway? Drill-down paths should be defined to slice the data into more reasonably-sized group/levels (web pages) to avoid that.
Regarding the error message itself, my guess is 50,000 rows of the data you're trying to display within a single web page, exceed some IIS memory buffer -- possibly a setting you can increase in IIS. I don't know. Or maybe its taking so long to stream that much data over the internet to the browser that something else interferes on the backend.
I've found that sometimes my carefully designed report in terms of drill-down path and styling just isn't suitable to do double duty: "let me interact with the data in a browser" -- versus -- "give me all the raw detail data so I can manipulate it further using a tool like Excel." In those cases I would create a separate report intended to download raw data. You could even schedule such a report to email a .csv file attachment. I know you said "Excel sheet" -- but I've never undestood why anyone would want the HTML-formatted data in their Excel sheet -- versus the underlying raw data, to then manipulate in Excel.
HTH.
Bob