Server-side script to pull stream from remote webcam

[RJM62]

I'm looking for ideas.

I'm building a site that features a live stream (sequential jpegs) from my webcam. It works, but the JavaScript on the page opens a new connection to my office PC for every visitor.

I'm think about building a server-side script that would pull the images and store them server-side, so the server will be the only computer connecting to mine. The site visitors would then pull them from the server.

I'm reasonably fluent in is PHP, but I'm not sure that's the best solution.

Any ideas?

Thanks,

--Rich

 

[Keane]

I would agree that your proposal is about as good of a solution as you could do. I would possibly build this 'caching' into your PHP script, so that it only updates the frame every few seconds.

Basically, when the user visits the PHP page, only grab the latest image from your webcam if the currently cached image is more than 5 seconds old or something.

That way your home computer isn't being put under too much stress, and you save network load if the image isn't requested for more than ~5 seconds.

 

[RJM62]

Thanks. I'm still in the "banging it around in my head" stage, but that's probably what I'll wind up doing.

-Rich

 

[Steve Foley]

I'm looking for ideas.

I'm building a site that features a live stream (sequential jpegs) from my webcam. It works, but the JavaScript on the page opens a new connection to my office PC for every visitor.

I'm think about building a server-side script that would pull the images and store them server-side, so the server will be the only computer connecting to mine. The site visitors would then pull them from the server.

I'm reasonably fluent in is PHP, but I'm not sure that's the best solution.

Any ideas?

Thanks,

--Rich

If it's a Windows Server, you can use Windows Media Server to serve up the stream, and Windows Media Encoder on the PC in your office. When someone requests the stream, WMS will open a connection to your PC and pull the stream. If a second user tries, they simply get a copy of the first stream.

I've done something similar with Linux using Darwin Streaming Server and an Axis Webcam.

 

[RJM62]

If it's a Windows Server, you can use Windows Media Server to serve up the stream, and Windows Media Encoder on the PC in your office. When someone requests the stream, WMS will open a connection to your PC and pull the stream. If a second user tries, they simply get a copy of the first stream.

I've done something similar with Linux using Darwin Streaming Server and an Axis Webcam.

Thanks. The home PC is running Windows (for now, anyway; I'm thinking about making an old laptop into a Linux server for it to save power). But the server is running Linux.

-Rich

 

[Steve Foley]

Thanks. The home PC is running Windows (for now, anyway; I'm thinking about making an old laptop into a Linux server for it to save power). But the server is running Linux.

-Rich

justin.tv may work, if you don't mind the stream running all the time.

 

[RJM62]

justin.tv may work, if you don't mind the stream running all the time.

I don't mind it running all the time, but this is a real low-priority site. I'm basically putting my pet turtle's tank online, and building a small educational site around it. It's something I haven't done before and there's no rush, so I'm looking to learn something in the process.

I have a commercial webcam server (BroadCam) actually providing the stream from my office PC, but I'm not married to it. It's just convenient for the time being.

Eventually I may dedicate an old laptop to the webcam and use one of the Linux streaming systems. For now, it's just easier to run it from this PC because my development tools live on this machine.

Basically, all I have to do is grab the output and cache it. But I may grab it and encode it on the Web side. I haven't really decided yet.

Oh, here's the turtle if you want to see it.

-Rich

 

[TangoWhiskey]

http://www.ustream.tv/

 

[RJM62]

Thanks, Troy. But like I said, I want to do this myself for the learning experience. For example, I've learned that the mini-webservers used by all of the webcam apps I've tried reject all WGET requests. The solution, I think, will be to install Apache on an old machine and dedicate it to running the cam server.

In other words, this is not a critical project. It's a learning / fun project. So thanks for the easy solution suggestion, but easy's not what I'm looking for right now.

-Rich

 

[TangoWhiskey]

IceCast will do it; I've used it for audio streaming, but not video--but it will do it.

http://icecast.imux.net/viewforum.php?f=6

 

[RJM62]

IceCast will do it; I've used it for audio streaming, but not video--but it will do it.

http://icecast.imux.net/viewforum.php?f=6

I may use it on the local end, but I can't use the relay on my web server. The server's running cPanel, and IceCast and cPanel have a conflict. Something to do with some library or another.

I will, however, scope out the code and see if I get any ideas. Thanks.

-Rich

 

[Areeda]

I haven't done anything with webcams (except set up one for my wife to skype with her family in Sweden).

Would you please describe what the software that runs the camera does? Does it save a bunch of static jpeg's or create some streaming video format? Does it have its own webserver or do you use Apache or IIS?

Just curious, I don't need a lot of detail.

Joe

 

[RJM62]

I haven't done anything with webcams (except set up one for my wife to skype with her family in Sweden).

Would you please describe what the software that runs the camera does? Does it save a bunch of static jpeg's or create some streaming video format? Does it have its own webserver or do you use Apache or IIS?

Just curious, I don't need a lot of detail.

Joe

Most USB webcams adhere to the UVC standard, so they have at least basic functionality without any special software at all being installed. UVC drivers are built into all modern operating systems.

One step up from that are the drivers typically provided by manufacturers, which may provide finer control or additional features that are not available using the standard UVC drivers.

The next step up are programs that do things such as take snapshots at specified intervals or when certain events (such as motion, sound, etc) occur. There are also monitoring programs that allow the camera to be seen (and something controlled) from remote locations, and streaming programs that allow the video signal to be viewed remotely.

I'm playing around with different proprietary and open-source programs, trying to find one I like. One that's pretty interesting is WebcamFLV, which captures the output and transcodes it to an FLV stream, which feeds a player stored on the server to produce a streamed Flash movie.

Most of the software I've looked at, however, "streams" sequential jpeg's, rather than an acual video format.

It's all pretty simple to set up, except for the relaying part. Most webcams only push still pictures or "videos" with framerates of less than 6 fps, which is pretty choppy. That's because of the limitations of FTP. HTTP pull allows for a better-quality stream, but the trick is how to cache it on the Web server so only one pull is made on the webcam server.

Most webcam streaming programs have small webservers built in that listen on a user-specified port for connections. This is fine for, say, a security camera, where only one person is likely to be connecting to watch it. For a publicly accessible camera, too many simultaneous connections could bog down the connection. So that's where some sort of relay or proxy is needed, so a webserver can pull the cam feed, and in turn feed it to multiple clients.

It's also possible, I'm told (though I haven't tried it), to assign the cam itself to a port, and then to map to that port using Apache, IIS, or another Web server; and then it would function pretty much as an IP cam. In this way, a server could pull the raw stream right from the camera, process it, transcode it, and serve it.

-Rich

 

[Areeda]

Thanks Rich.

I've been toying with the idea of some security cameras around the house.

Joe

 

[RJM62]

Thanks Rich.

I've been toying with the idea of some security cameras around the house.

Joe

There are plenty of ready-made programs for that. Some are included with specialized cameras and systems that are made for that purchase and which are not very expensive. Some software to do the same with an ordinary webcam can be purchased on the cheap or even downloaded for free on the Web.

I actually had my turtlecam online almost a week ago. That was the easy part. Trying to use my web server to relay the stream is harder -- and frankly, probably unnecessary. I doubt I'll be overwhelmed with people who want to watch a turtle basking on a plastic rock.

But I've learned quite a bit in the process of trying, which really is my objective more so than simply putting a turtle tank on the Web.

-Rich

 

[jesse]

There are plenty of ready-made programs for that. Some are included with specialized cameras and systems that are made for that purchase and which are not very expensive. Some software to do the same with an ordinary webcam can be purchased on the cheap or even downloaded for free on the Web.

I actually had my turtlecam online almost a week ago. That was the easy part. Trying to use my web server to relay the stream is harder -- and frankly, probably unnecessary. I doubt I'll be overwhelmed with people who want to watch a turtle basking on a plastic rock.

But I've learned quite a bit in the process of trying, which really is my objective more so than simply putting a turtle tank on the Web.

-Rich

It looks like it is offline right now. Whenever it is online again I can take a look at the javascript and how it works and likely point you in the right direction for writing a caching layer on the server.

 

[RJM62]

It looks like it is offline right now. Whenever it is online again I can take a look at the javascript and how it works and likely point you in the right direction for writing a caching layer on the server.

Thank you very much.

http://www.myturtlecam.com/

-Rich

 

[wsuffa]

There are plenty of ready-made programs for that. Some are included with specialized cameras and systems that are made for that purchase and which are not very expensive. Some software to do the same with an ordinary webcam can be purchased on the cheap or even downloaded for free on the Web.

Take a look at the capture boards from QSee. They work pretty well... but use analog cameras. For a few hundred $$ you can have an 8 camera security server system running on a PC. Even works pretty well on an old copy of Win2000 (put it behind a firewall with only the necessary ports open, or even better use a VPN.

 

[RJM62]

It looks like it is offline right now. Whenever it is online again I can take a look at the javascript and how it works and likely point you in the right direction for writing a caching layer on the server.

Jesse, most of the JS on that page was lifted from the network version of the page generated by the BroadCam application. The one the application generates works fine, but has a lot of extra junk that's required if you have an unlicensed copy. I have a licensed copy, so I was able to remove it.

I also made a few changes to the JS (re-pointing the source to the zapto.org subdomain rather than the IP, adding the code to replace the stream with the offline image when the local server is down, adding the popup page, etc.), but most of the meat was generated by the application itself.

There's also another version of the page that uses a Flash stream, which I cannot get to work over the network. But the one generated by the application doesn't work, either, even when pasted intact. But I don't care for the quality of the Flash version, anyway, so I just disabled it.

I have been trying for several days to figure out how to tap into the stream itself, without much success. The closest I came was when I mapped the output of another streaming program to a WebDAV drive on the web server, which actually worked, but choppily. I haven't completely given up on that option, but...

Last night, before I read your post, I started looking into caching proxies as another option. I was looking at source code from Sourceforge and Freshmeat until 3 a.m. Your idea would be a lot easier, and I appreciate any suggestions along those lines.

As I said earlier, this is intended to be a learning / fun project, and so far it's been both. This is a mid-life career change for me and I have massive gaps in my knowledge, and this is how I like to fill them.

Thanks again,

-Rich

 

[RJM62]

I think I finally identified the crashiness problems with the BroadCam software. It ran pretty well on this machine (homebuilt Atlhlon 64X2 with 2 GB system RAM and 512 MB video RAM), but not on any of the others I tried. It seems BroadCam is just a bit of a hog, pure and simple.

It was consuming 100 percent of processor resources on a couple of older box I'd dedicated to it, the most recent being an old Dell 2300 with a P4 and a gig of RAM. The application hung or crashed several times today on that machine. So I decided to run it from a Vista 64 box that I don't use for very much else; and once I got past all the UAC nonsense, it installed and seems to be running well. I may try the Flash output again in the morning.

In any case, if you're thinking about Webcam streaming software and BroadCam is one of your choices, be aware that it's a hog.

-Rich

 

[RJM62]

After looking over code from a number of open-source projects to see how they did things (most notably Motion and Red5), I did finally come up with a proxy / distribution system that worked. Several of them, actually, including using Motion (resource-expensive) and Red5 (resource-expensive plus it kept crashing).

But the one that worked both best and worst in the end worked by connecting my netcam to the web server over the Internet at OS level so it would get the raw MJPEG stream, and then doing all the processing on the server. I was able to use FFMPEG to convert the stream to pretty much anything I wanted (flv, mpeg, wmv...) or just leave it as an MJPEG. I then used a variety of embedding techniques (JavaScript, Flash, Cambozola, etc.) to place it on the page.

The problem with this was that uploading the raw MJPEG it required a stable data stream and consumed a lot of bandwidth between the server and my office, and periodically the available bandwidth would be reduced when I was using the connection for other things, which would crash the whole shebang. I tried using variable bitrates, but that resulted in the framerate going down to as low as one-point-something at times.

Then I used my router's QOS functionality and played with the image quality and framerate at the camera side. I found a compromise that stabilized things, but made the final video quality marginal. Still, it was marginally acceptable, so I decided to let it run overnight with the browser window open on my end to see if and how well it worked.

By this morning, the video was still going, with an average framerate of nine-point-something and a range from five-point-something to twelve-point-something. Not great, but marginally acceptable. And the system hadn't crashed.

But then I looked at the bandwidth consumed (server-side) and it was 1.79 GB over something like nine hours. Unacceptable. I have excess bandwidth server-side, but my local cable provider would have fits if I ran that 24/7.

So in the end, my free solution was too expensive.

I haven't totally given up: Red5 looks promising (although it repeatedly crashed on my server). I could then do the FLV conversion here, which would reduce bandwidth requirements. But it seems resource-wasteful for my actual purpose.

So right now I'm leaning toward one of the many free or commercial services that are already out there. I satisfied my primary objective, which was to learn how to do this myself. I made it work in a number of different ways, and learned marketable skills in the process. If somebody, for example, wants a multi-camera, IP-based surveillance system installed, I can easily set it up at this point using Motion at 1 FPS except when an event is detected, and then jump it up to, say, 10 FPS and record the event.

So in the end, I accomplished what I wanted to to. I learned something. But for my actual site, I'm leaning toward a commercial solution. Running Red5 on my server to serve up one video of two turtles in a tank seems like a wasteful "solution" to me.

-Rich

 

[RJM62]

Well, I changed my mind again.

I checked out the free services, and all except one plasters overlay ads on the videos (in addition to pre-rolls, in most cases). No good. I understand they have to support the services, but ads on the videos themselves is just too... tacky.

Most of the paid services charge more than it would cost for me to lease another server and install FMS on it.

So I decided to encode the webcam stream as an FLV using a neat little freeware program, and then built a reverse proxy in Apache. Seems to work: I had four simultaneous connections showing up as one and pulling < 30kbs, so I guess it's working.

The only drawback is that I have to have a Windows computer running doing little other than creating the FLV feed. I would just run it on this one, but very often I use proxy servers, which throws the dynamic IP service off.

Here's the feed:

http://www.myturtlecam.com/liveFeed/

I'm still tweaking, trying to find the sweet spot for quality vs bandwidth. But at least it works. Now I can get around to building the site...

-Rich

 

[TangoWhiskey]

Seems to work fine from my end! Chrome shows it just fine. Nice job, great learning experience. Too bad stuff like this is still so complicated!

P.S. I tuned in at the right time. Your turtle is off the rock and swimming!

 

[RJM62]

Thanks. I really did it for the learning experience, and in the end, the simplest way was the best.

There are three turtles in there, but two are only like an inch long.

The lights are on a timer. I can override it and keep them up a little later if you like.

-Rich

 

[jesse]

Looks good. It's amazing what one can build and learn when they put their mind to it. Sometimes using pre-packaged commercial stuff makes the most sense - other times it's better to just learn it. In the end, you're smarter and more valuable.

 

[RJM62]

Thanks, Jesse. I went through many different packages -- Motion, Darwin Streaming Server, Red5... probably a dozen others. In the end, a simple reverse proxy worked best. I still have a few things to tidy up server-side and some tweaking to do with the Flash settings, but I think this will be the solution I'll stick with.

I also learned that red night lights work better than blue ones as far as the camera is concerned. I'll have to watch the turts, though, to make sure it doesn't keep them awake. I'm not sure they can see red. I'll have to research it.

I'm really enjoying these little guys, lol. They have such different personalities. One doesn't like to be handled (but will eat out of my hand), one is indifferent, and one swims right on to my palm if I lay my hand in the water. Lotsa fun.

-Rich

 

[TangoWhiskey]

I'm really enjoying these little guys, lol. They have such different personalities. One doesn't like to be handled (but will eat out of my hand), one is indifferent, and one swims right on to my palm if I lay my hand in the water. Lotsa fun.

-Rich

Which personality is the large turtle?

 

[RJM62]

Which personality is the large turtle?

The larger one is about an inch and a half in carapace length, and he (she?) doesn't care for being handled. When I have to remove it from the tank, it tries to swim away from me. But it will eat out of my hand.

Of the two smaller ones (they're about two weeks younger), one is pretty much indifferent and doesn't seem to mind being handled, but doesn't seek it out, either. The other loves to be held and swims right into my palm.

One of the two turtles I got in the first shipment didn't make it through the night. It just got too cold during shipping. We had a real cold snap that week, and the UPS guys around here like to drive around with all the doors open to save time.

I was worried about it from the start because it was so cold when I took it out of the package, and the next morning it was dead. So the breeder sent me two replacements, by overnight UPS. They were perky and energetic as soon as I opened the package.

-Rich

 

[dc-soar]

I'm hoping you will share the solution you found for setting up your turtles Live FLV reverse proxy server.


I've also tried some of the packages you tried, and even tried "Google's: Youtube live streaming", but they all have advanced features that prevent fully automating an old laptop at the receiving end, (or they have ads).


For the little church I volunteer at, I was building a site that had a live stream of the services for our elderly shut-ins.
I'm using recycled old laptops for the shut-ins receivers, and it really brightens their week seeing and hearing their church family twice a week.


But the elderly find using feature rich technology very very stressful.


I've tried showing just what to click, leaving them with a printed recipe of what to click, and walking them through by phone what to click. These work, but I'd prefer a FLV file name-path that is the same week to week, so the receiving laptops can just turn on running the stream with no clicks, as a single use media streaming brick.


I've set up many simple DNS and simple Apache servers in the antique days of Linux, reasonably fluent in a few programing languages, and done some Tomato router scripts, routed ports for cameras in its firewall, and configured Dyn DNS.


Your solution seems elegantly simple.
But I've found it easy to waste days looking for a missing “=” in a “==” in a script, or a missing comma.

Please save me from “reinventing the wheel”.

-Thanks David

 

[radio]

How about writing a program (or just using a .bat and a sched task) to FTP the latest jpg to your server every n sec? Then have the webpage just display the latest image. Use JS/Jquery to check if the image has been updated, and then reload it client-side.

 

[RJM62]

I'm hoping you will share the solution you found for setting up your turtles Live FLV reverse proxy server.


I've also tried some of the packages you tried, and even tried "Google's: Youtube live streaming", but they all have advanced features that prevent fully automating an old laptop at the receiving end, (or they have ads).


For the little church I volunteer at, I was building a site that had a live stream of the services for our elderly shut-ins.
I'm using recycled old laptops for the shut-ins receivers, and it really brightens their week seeing and hearing their church family twice a week.


But the elderly find using feature rich technology very very stressful.


I've tried showing just what to click, leaving them with a printed recipe of what to click, and walking them through by phone what to click. These work, but I'd prefer a FLV file name-path that is the same week to week, so the receiving laptops can just turn on running the stream with no clicks, as a single use media streaming brick.


I've set up many simple DNS and simple Apache servers in the antique days of Linux, reasonably fluent in a few programing languages, and done some Tomato router scripts, routed ports for cameras in its firewall, and configured Dyn DNS.


Your solution seems elegantly simple.
But I've found it easy to waste days looking for a missing “=” in a “==” in a script, or a missing comma.

Please save me from “reinventing the wheel”.

-Thanks David

Hi David,

Sorry I missed your post. It's been a bit hectic the last few days.

When all was said and done with the turtle cam, this is what I used:

http://sourceforge.net/projects/webcamflv/

It was nice to cuss and swear about learn the intricacies of Linux streaming, but it was even nicer to install a simple program on an old Windows XP laptop in my office, and just let it run. I notice they have updated the program since I installed it; maybe I'll play with the new one later.

I don't know how your church is set up, but probably you should set up a reverse proxy on a Web server to cache and serve the stream if you plan to have more than a few people watching.

-Rich

 

[dc-soar]

Thanks RJM62,
I've taken a quick look at webcamFLV and it looks CLEAN an SIMPLE no extras and no kitchen sink, everything running from my server, no back path handshakes.

Most everything I've tried so far worked, if you had computer literate people at both ends. But because of their advertising containers, trying to get them to do the last simple 1% tweak I needed for elderly users, felt like i was jumping through too many hoops and I was just banging my head against a wall.

I hope to get time tomorrow, to give it a real try.

Thanks again,
David