Max Content Size?

[ThraxxMedia]

Hi everyone,

I was just wondering about the issue described in the title - apparently there's an internal limitation in Replay Media Catcher which only grabs HTTP stream content up to a maximum size of 2,097,151 KB (approx. 2GB).

Having a look at the respective log reveals the following:


Download file: a2f412b4-6e81-4b12-b54e-0219d009ea12.flv
Started at: 23.10.2011 17:54
Queued for downloading
HTTP-Download gestartet ...
Dateiname ermittlung ...
Applying HTTP Page Title naming rule
Dateiname von Seite-Titel entnehmen ...
Dateinameerkennung abgeschlossen
Der Download ist 2097151 KB groß...
Content length exceeds max size (2147483647 > 2147483647). Performing a single segment download.
Download von Segment 0 wurde gestartet

I apologize for the random German parts in it, but that seems to be another kind of bug in your software. Language is properly set to English, still it shows up like the above.

Anyway. I have highlighted the important line where it says that content length is exceeding max size, and thus performing a single segment download. Now this is kind of stupid, because I am operating at Windows 7 x64 with NTFS file system, so I should normally be able to create files larger than 4GB.

The way the HTTP stream I'm trying to capture is saved, is the following:

It's a constant, single stream (from start to finish, roughly 10 hours per day over 2 days of content), and you may jump to several parts of the stream by picking a choice from a website, which then simply sets the current stream pointer (seek position) to a given amount of seconds.

That means, if I pick the very first part of the show (stream pos 0), RMC grabs the first 2GB from position 0.
If I pick any other part of the show, lets say jump to position 1800 (which is 30 minutes from the beginning), it again grabs only 2GB, but now 30 minutes offset.

With additional effort, I was able to capture portions of the stream, but I came across the following problems:

1. Every segment is always 2GB in size, EVEN if the chosen part of the show lasts only a couple of minutes. There's no way RMC could recognize the proper end position, which would simply be the starting position of the next segment.

2. Consequently, if one part of the show exceeds the duration of 2 hours, 22 minutes (which roughly equals the 2GB limit), it's only captured partially. There's just no way I could jump to positions other than the ones given by the website.

2. The downloaded 2GB segments are broken. I have to use the "Fix FLV"-Tool so that the duration shows up properly and I may load it into another program for further editing. This is nothing serious, it's ok since the "Fix FLV"-Tool does a fine job. But it's time consuming and an unnecessary step, imho, if it would save the segments correctly from the beginning.

You may see for yourself and have a look at it here:

http://eu.battle.net/blizzcon/en/live-stream/

It's the VOD of BlizzCon 2011; some parts may require a virtual ticket, but some parts (I believe the RTS and WoW tournament feeds) are free to watch.

On a side note: the durations displayed on their website are all wrong. You'd have to calculate the proper durations for each part of the show individually by looking at the seek positions. You can get the actual seek position of one part from the link that shows in RMC log. If the stream just starts from the beginning, this is dropped and not included in the link.

Here's a good example (of course free to watch):

http://blizzcon2010-f.akamaihd.net/blizzrtsday1_1_2000@s36303?v=2.4.7&fp=WIN%2011,0,1,152&r=DKBXZ&g=UNNITUTORERT&seek=37800&primaryToken=1319417612_f97464b2c24b372de442953408 9c326f

Notice the url part where it says "seek=37800"; this is listed on the website as "StarCraft II GSL Finals", and is in fact a segment which exceeds the 2GB limit.

I would appreciate if you could help me in any way.
I'm using the current version (4.3.2) of RMC, but I have also tried 3.11 with no luck. Same limitation there.

Oh and you'd better hurry since the stream is only available until 06. Nov 2011.

Thanks for your time.

[Cela]

I am a beginner. But from my experience I do not think there is a download limit within Replay Media Catcher 4, at least not at 2GB. Though, there may be limits elsewhere, e.g. imposed by the sender, your internet provider or on your own computer (available free disk space, format of hard disk).
Looks like a limitation of disk format of your storage folder?
FAT16: files are limited to max. 2GB.
(Where is my storage folder ? See >Tools> Settings> Download Settings > Storage Folder)
Try a disk formatted as NTFS. I'm not quite sure, FAT32 may work too.
I use a NTFS formatted disks, and mostly downloaded 2GB+ files (largest so far was ~3.6GB).

What do the experts say?

[ThraxxMedia]

If you'd have read the entire post above, you'd know that I already clarified this:

I am operating at Windows 7 x64 with NTFS formatted hard disk drives ONLY, so I won't EVER run into a disk format issue regarding file size.
Also, I am NOT using any kind of web browser for download attempts (I mention this because there's a known issue with download size limitation on older versions of IE).

On top of that, I am definitely NOT as dumb as you're trying to address me here. I am perfectly able to check free hdd space on my own, and when I tell you there is PLENTY of it, then it shouldn't even remotely come into question.

Just believe me when I'm saying there simply is no option left which could possibly set that limit on MY side, so I can only imagine two other reasons:

1. Limitation by sender. I wouldn't exclude it 100%, BUT the file name of the stream is ALWAYS the same for each segment (figured that by checking the url), which implies it must be a single, huge file where you're just jumping to different seek positions.

2. Limitation by RMC.

Judging by what I see, I would pick option 2. And that's exactly why I made this topic.

[Jeff Lenney]

I'm looking into this now, I will report back once I have more info.

Thanks!

[Jeff Lenney]

Hi

This is not an issue with our software.

The HTTP header from the server has returned the content-length as 2GB.

For such large downloads >= 2GB we fall back from a multi-segement download to a single segment download as it is so large that a multi-segment download will run slower than a single. Live HTTP streams also report a content-length of 2GB.

I dont have access to the site (it states access denied) but it sounds like from your comments Id say they have broken then VOD stream up in to 2GB segments.

You would need to start to play each segment to download.

This is just straight HTTP so what ever is downloaded by browser - we download it.

As for the language - the download log is not translated at display time. Once it is written it stays in that language (its just a text file on disk).

[ThraxxMedia]

Hi there,

thanks so far for your reply. Please let me add the following detail:

You may be right about the VOD stream being split up in some way - I'm not claiming I'd have any viable knowledge to say otherwise - but still I think there are no fixed segments, because you may jump to ANY position on the stream, resulting in a 2GB downloaded file, starting from the exact position of your choice. So lets say for example, you have the complete show with a fixed length of 5 hours flat. That show consists of 3 parts:

The 1st part starting at 00:00, ending at 00:30,
thus the 2nd part starting at 00:30, ending at 02:42,
the 3rd part starting at 02:42, ending at 05:00 (end of stream).

Also, lets say 2GB of video equal to 02:22 play time (at least that's how it is for the stream I'm trying to capture).

Now, the way it works out is like this:

If you start watching the stream (and capturing) from the very beginning, at 00:00, it will download a 2GB file with a play time ranging from 00:00 to 02:22, which covers the 1st AND, partially (with 20 seconds missing), 2nd part of the show.

If you start watching from 00:29, it will download a 2GB file with a play time ranging from 00:29 to 02:51, which covers exactly 1 second leftover from 1st part, the whole 2nd part AND an additional 9 seconds from the 3rd part.

So this is kind of stupid. The stream doesn't seem to have fixed segmentation, because if it would, then shouldn't jumping to a position which is close to the end of a segment result in a smaller file (because of, well... reaching the end of segment)?

I am very confused about the issue.

On a side note: the only case where RMC actually DOES download a file smaller <2GB is when jumping to a position close to the absolute end of the whole stream (rather than end of a part in mid-stream) - obviously, because the stream ends and there is no more video data to retrieve.

Unfortunately, I don't know of any other website using that kind of streaming technology. Still, I hope I could shed some light on it here.


The HTTP header from the server has returned the content-length as 2GB.
Is there any possibility the HTTP header could return otherwise, or maybe methods to work around that? Perhaps HTTP content can never be larger >2GB - this would be a pity, but kind of the explanation I was looking for.

[Cela]

...On top of that, I am definitely NOT as dumb as you're trying to address me here....so I can only imagine two other reasons:

1. Limitation by sender. I wouldn't exclude it 100%, BUT the file name of the stream is ALWAYS the same for each segment (figured that by checking the url), which implies it must be a single, huge file where you're just jumping to different seek positions.

2. Limitation by RMC.

Judging by what I see, I would pick option 2. And that's exactly why I made this topic.It is a great topic and I simply wanted to help you by sharing my experiences!

But I am able to return in your impolite tune:
If you'd have read my entire post above, you'd know that I already clarified this:
Common sense deducts:
1. Limitation by sender.
2. Limitation at your side.
3. NO 2GB limitation by RMC, since I can download from my sender to my site at least 3.6GB which is definitly larger than your 2GB.

So I am the lucky one. You are ... impolite!

[Jeff Lenney]

Hey Folks

Please keep it civilized or i'll lock the thread. Thanks!

[Jeff Lenney]

Hi there,

thanks so far for your reply. Please let me add the following detail:

You may be right about the VOD stream being split up in some way - I'm not claiming I'd have any viable knowledge to say otherwise - but still I think there are no fixed segments, because you may jump to ANY position on the stream, resulting in a 2GB downloaded file, starting from the exact position of your choice. So lets say for example, you have the complete show with a fixed length of 5 hours flat. That show consists of 3 parts:

The 1st part starting at 00:00, ending at 00:30,
thus the 2nd part starting at 00:30, ending at 02:42,
the 3rd part starting at 02:42, ending at 05:00 (end of stream).

Also, lets say 2GB of video equal to 02:22 play time (at least that's how it is for the stream I'm trying to capture).

Now, the way it works out is like this:

If you start watching the stream (and capturing) from the very beginning, at 00:00, it will download a 2GB file with a play time ranging from 00:00 to 02:22, which covers the 1st AND, partially (with 20 seconds missing), 2nd part of the show.

If you start watching from 00:29, it will download a 2GB file with a play time ranging from 00:29 to 02:51, which covers exactly 1 second leftover from 1st part, the whole 2nd part AND an additional 9 seconds from the 3rd part.

So this is kind of stupid. The stream doesn't seem to have fixed segmentation, because if it would, then shouldn't jumping to a position which is close to the end of a segment result in a smaller file (because of, well... reaching the end of segment)?

I am very confused about the issue.

On a side note: the only case where RMC actually DOES download a file smaller <2GB is when jumping to a position close to the absolute end of the whole stream (rather than end of a part in mid-stream) - obviously, because the stream ends and there is no more video data to retrieve.

Unfortunately, I don't know of any other website using that kind of streaming technology. Still, I hope I could shed some light on it here.


Is there any possibility the HTTP header could return otherwise, or maybe methods to work around that? Perhaps HTTP content can never be larger >2GB - this would be a pity, but kind of the explanation I was looking for.

Hi - The HTTP response header from the server has content-length field set to 2GB.

This means that the server is telling us that the data it will return is 2GB in size. If it returned more than 2GB then RMC/Jaksta would keep on downloading in any case.

In other words:

1. Someone at that site has configured their server so that when a request comes in it will return 2GB (max by the looks of it) of data in any one request.
2. Your browser has made a request and the server has responded telling us that 2GB of data will be returned.
3. RMC says wow thats a lot of data so lets just use one connection for that request and starts downloading until the server stops sending. This is the same request as what made by the player in the browser.
4. The server stops sending after it has sent 2GB of data just like it said it would.

Whether the server has one large file and sends it out in 2GB at a time or it has many 2GB files, is invisable to us. All as we know is that a request has been made and the server is telling us that 2GB will be returned. We then keep downloading until the server says that there is no more data.



Hope that helps.

[ThraxxMedia]

Thank you very much. Even if it doesn't really solve my issue, but let alone the information was kind of helpful. I think I'll get "around" the limitation by just downloading the 2GB segments, appending them, and then break up the resulting file into parts as needed (with another software).

Again, thanks for hearing me out.

As for the other guy... no comment on that one.
Cheers!

[Cela]

As for the other guy... no comment on that one.
Cheers!He wasn't that wrong:

Common sense deducts:
1. Limitation by sender.
2. Limitation at your side.
3. NO 2GB limitation by RMC...Jeff Lenney, Technical Support, confirmed and explained "1. Limitation (imposed) by sender."

Would "Always record streams instead of downloading" yield a solution?
I doubt it.
When I tested that option, RMC4 recorded for me some 32 minutes from a 2 hours 46 minuts stream into a "We're sorry... .flv" file and then reported "Finished". For that recording/download test the normal "Download" mode downloaded with 20Mbit/s download speed the full flv.
In the User's Guide I would expect more details about how to succeed with tricky recordings.

Cheers!

Disclaimer:
I am new to RMC4. I do not intend to offend anybody's expertise. I simply want to share my limited knowledge.
I may have overlooked something in some postings which may make my posting redundant or obsolete. If so, simply neglect it.