The `raspivid` tool grabs raw H.264 compressed video. Unfortunately, this stream is lacking frame timestamps. If you're just grabbing the default 25 frames per second and the system never stutters and always grabs frames at exactly that rate, it's fine. I was playing around with grabbing at 10 fps for a lower bitrate stream and that was confusing all the tools. ffmpeg considers a raw H.264 stream without timestamps to be a bug and the warning messages suggests that they won't support importing it in the future.
raspivid does have a way of capturing frame timestamps to a separate text file, and mkvmerge can reintegrate that into a video file. From there, ffmpeg can convert that '.mkv' file to the more popular '.mp4' file format.
sudo apt-get install -y mkvtoolnix ffmpeg
echo "# timestamp format v2" > vidf.264.pts
raspivid -o vidf.264 -t 10000 -pts -n -fps 10 >> vidf.264.pts
mkvmerge -o /tmp/b.mkv --timestamps 0:vidf.264.pts vidf.264
ffmpeg -i /tmp/b.mkv -c copy /tmp/f.mp4
mkvmerge needs a header line on the frame timestamp file, so we create the file with just that header, then append the data to it. The raspivid command captures 10 seconds (10000 milliseconds) at 10 frames per second with no on screen preview. It captures the video to the named file vidf.264 and writes the frame timestamps to standard out which is appended to vidf.264.pts . mkvmerge then does the reintegration of the timestamps with the video stream. Lastly, ffmpeg converts the mkv file (which works directly with VLC and other viewers) to mp4 which works with just about everything.
Friday, September 21, 2018
Tuesday, July 10, 2018
Wikipedia walk: Asteroids
https://en.wikipedia.org/wiki/Asteroid
Asteroids, big rocks floating around our solar system. Sometimes they hit Earth and kill the dinosaurs. The name derived from 'aster' (star) 'oid' (like), because they were star-like in being points of light in the sky that didn't resolve into circles with the telescopes of the era in which they were identified, while those telescopes did show noticably circular planets and the rings of Saturn. But they're not stars, because stars don't move (on human time scales).
With varying quality we have data on 504,000 to 780,000 asteroids. We're discovering and tracking an additional 2000 near-Earth objects every year.
Telescopes and Surveys. We have a couple telescopes doing non-stop survey work of the sky in Arizona and Hawaii, there are and have been others, and the LSST in Chile, coming online around 2021, looks awesome. These surveys image the whole sky every few days and software can automatically check, "hey, did that speck of light move?". If yes, it might be an asteroid. These projects are already moving "big data" with substantial storage and computation needs, and the LSST is going to take that up a big notch.
I think it's cool that we have telescopes devoted full time to scanning the sky, and big data backing them up doing analysis. This kind of research makes a bunch of our sci-fi future possible. They have short lists of a few dozen asteroids a relatively short hop from Earth that we could mine. They have lists of asteroids that might impact Earth that we can keep an extra close watch on them so that we don't go the way of the dinosaurs. From the apparent distribution of objects in our solar system, we probably know about over 90% of the 1km+ objects that might hit earth, and better technology and techniques are pushing that coverage higher and object size threshold lower (future goals of a couple projects are all the objects over 140m or over 30m). All this for a global budget of just a few tens of millions of dollars. We can do this. When we have it together, humanity is awesome.
Asteroids, big rocks floating around our solar system. Sometimes they hit Earth and kill the dinosaurs. The name derived from 'aster' (star) 'oid' (like), because they were star-like in being points of light in the sky that didn't resolve into circles with the telescopes of the era in which they were identified, while those telescopes did show noticably circular planets and the rings of Saturn. But they're not stars, because stars don't move (on human time scales).
With varying quality we have data on 504,000 to 780,000 asteroids. We're discovering and tracking an additional 2000 near-Earth objects every year.
Telescopes and Surveys. We have a couple telescopes doing non-stop survey work of the sky in Arizona and Hawaii, there are and have been others, and the LSST in Chile, coming online around 2021, looks awesome. These surveys image the whole sky every few days and software can automatically check, "hey, did that speck of light move?". If yes, it might be an asteroid. These projects are already moving "big data" with substantial storage and computation needs, and the LSST is going to take that up a big notch.
I think it's cool that we have telescopes devoted full time to scanning the sky, and big data backing them up doing analysis. This kind of research makes a bunch of our sci-fi future possible. They have short lists of a few dozen asteroids a relatively short hop from Earth that we could mine. They have lists of asteroids that might impact Earth that we can keep an extra close watch on them so that we don't go the way of the dinosaurs. From the apparent distribution of objects in our solar system, we probably know about over 90% of the 1km+ objects that might hit earth, and better technology and techniques are pushing that coverage higher and object size threshold lower (future goals of a couple projects are all the objects over 140m or over 30m). All this for a global budget of just a few tens of millions of dollars. We can do this. When we have it together, humanity is awesome.
Saturday, May 26, 2018
CPU MFLOPS Test in 6 Languages
Once upon a time my dad worked on supercomputers at Los Alamos National Laboratory and sometimes he would spend a minute running a little test to see about how fast a new supercomputer was. I kept that test, flops.c, very much as I received it in 1992. Your phone is probably much faster than the fastest supercomputer of 1992.
This isn't a very good benchmark, but it's simple and easy. It runs a few basic numeric algorithms with a known number of adds, subtracts, multiplies, and divides, and figures out how many floating point operations per second your CPU can do. It doesn't take advantage of multi-core or SIMD instructions. It doesn't exercise the memory system and probably all fits in L1 cache. It just tests how fast your CPU can do math (and that your compiler isn't terrible at making that happen).
Over the years I transliterated flops.c into other languages to test them and their compilers and interpreters. Python has a pretty slow interpreter, around 1-5% of the speed of C. Javascript got amazingly good and can run 80-90% of the speed of C. Java got to that speed around 2007 or 2008. The Go compiler is surprisingly good for a newer language, it knows some tricks GCC doesn't. Julia is a newer language that should have the potential for running full speed but apparently still needs some tweaking (as of 2018-05).
Note that Go and Julia have some compiler trick the rest are missing. Test 2 is to calculate Pi from atan(1.0) by Taylor series. Without reading the assembly I'm guessing there's some fused-multiply-add going on at least. GCC 7.3.0 with -O3 didn't get it. Go reports 147 Gigaflops on test 2, and Julia 22 Gigaflops. Everything else is showing my cpu in the 5-6 Gigaflop range.
Sources on github:
https://github.com/brianolson/flops
This isn't a very good benchmark, but it's simple and easy. It runs a few basic numeric algorithms with a known number of adds, subtracts, multiplies, and divides, and figures out how many floating point operations per second your CPU can do. It doesn't take advantage of multi-core or SIMD instructions. It doesn't exercise the memory system and probably all fits in L1 cache. It just tests how fast your CPU can do math (and that your compiler isn't terrible at making that happen).
Over the years I transliterated flops.c into other languages to test them and their compilers and interpreters. Python has a pretty slow interpreter, around 1-5% of the speed of C. Javascript got amazingly good and can run 80-90% of the speed of C. Java got to that speed around 2007 or 2008. The Go compiler is surprisingly good for a newer language, it knows some tricks GCC doesn't. Julia is a newer language that should have the potential for running full speed but apparently still needs some tweaking (as of 2018-05).
Sources on github:
https://github.com/brianolson/flops
Monday, August 21, 2017
2017 eclipse Raspberry Pi pinhole box
I recorded the 2017 eclipse from where I was in Boston using a pinhole box with a Raspberry Pi 3 and 8MP camera in the box.
Pinhole at top left, display screen at bottom left, rpi, camera, and battery.
Here's the resulting video, 10 seconds per frame displayed at 30 frames per second for a 300x speedup.
It never quite focused right. If I were to try this again (in 7 years, with a rpi version 9 and whatever upgraded camera comes with it then) I'd try harder to seal off the light leaks from the box. I wasn't on my usual network and couldn't log into the rpi to get images as it went so I had to open the box now and then to reposition and track the sun.
Here's the resulting video, 10 seconds per frame displayed at 30 frames per second for a 300x speedup.
It never quite focused right. If I were to try this again (in 7 years, with a rpi version 9 and whatever upgraded camera comes with it then) I'd try harder to seal off the light leaks from the box. I wasn't on my usual network and couldn't log into the rpi to get images as it went so I had to open the box now and then to reposition and track the sun.
Wednesday, October 12, 2016
Postgres global locking problem
It seems that PostgreSQL can't `VACUUM FULL` one database and `pg_dump` another database at the same time. These things were going on in a database I was observing and I saw things back up waiting for an exclusive lock on pg_catalog.pg_db_role_setting. I found in the documentation that `VACUUM FULL` is one of the only things that takes out such an exclusive lock such that nothing else can even do a read-only `SELECT` off the table that's locked. The log file noted "waiting for AccessShareLock on relation ..." and a oid number which turned out to be pg_catalog.pg_db_role_setting.
So, that's annoying, and a reason that we're going to have to continue splitting up postgres servers into more single-function servers, because there isn't actually enough internal isolation sometimes.
Monday, April 25, 2016
Democracy Still Needs Fixing
Hundreds of thousands of people wanted to vote in the primaries this year, but were denied by the parties, and we let this happen. We let private clubs determine if someone can vote. Isn’t that absurd?
I’d kinda like to get rid of publicly funded primaries. Why do we spend millions of dollars of public money to run a private club’s organizational internal vote?
Maybe we make a deal, in exchange for public funding of the primary vote, the party has to let anyone vote in their primary (regardless of registration in that or any other party). If the X party want a closed primary, they can run it (and pay for it) themselves. Maybe the Internet Party will have an online primary, that’d be cool, eh?
Furthermore, people should be insulted that they didn’t get to vote in the primary because a Party didn’t want their vote, didn’t want their participation. The parties should be worried that they’re insulting and driving away people. Except that actually they both seem happy to play the game of trying to disenfranchise as many voters as possible (who might possibly vote for the other side).
I want the answer to be more democracy, not less. We could break the lock held by The Two Parties if we changed ballot access laws and used a rankings or ratings ballot in the general election. A ‘primary’ would be a private function of a party that gets a slot on the ballot because it has some number of registered members. Anyone else can get on the ballot by signature petition. On the ballot you get to rank your choices 1st, 2nd, 3rd, etc…; or rate them on a scale of 0..10. Our current focus on primaries is an artifact of needing to get down to two choices on the final ballot, but that’s not necessary with a better ballot where you can express yourself on the whole field (or as much of it as you care to).
Break the two party duopoly. Another rant, because it’s Monday.
I’d kinda like to get rid of publicly funded primaries. Why do we spend millions of dollars of public money to run a private club’s organizational internal vote?
Maybe we make a deal, in exchange for public funding of the primary vote, the party has to let anyone vote in their primary (regardless of registration in that or any other party). If the X party want a closed primary, they can run it (and pay for it) themselves. Maybe the Internet Party will have an online primary, that’d be cool, eh?
Furthermore, people should be insulted that they didn’t get to vote in the primary because a Party didn’t want their vote, didn’t want their participation. The parties should be worried that they’re insulting and driving away people. Except that actually they both seem happy to play the game of trying to disenfranchise as many voters as possible (who might possibly vote for the other side).
I want the answer to be more democracy, not less. We could break the lock held by The Two Parties if we changed ballot access laws and used a rankings or ratings ballot in the general election. A ‘primary’ would be a private function of a party that gets a slot on the ballot because it has some number of registered members. Anyone else can get on the ballot by signature petition. On the ballot you get to rank your choices 1st, 2nd, 3rd, etc…; or rate them on a scale of 0..10. Our current focus on primaries is an artifact of needing to get down to two choices on the final ballot, but that’s not necessary with a better ballot where you can express yourself on the whole field (or as much of it as you care to).
Break the two party duopoly. Another rant, because it’s Monday.
Saturday, February 13, 2016
Ruby Scope
Ruby has perhaps a new variation on the bad old LISP thing of some functions evaluating in the context where they're called rather than in a self-contained scope.
class Foo
def initialize(a)
@a = a
end
def do_wat(m)
wat_helper(m)
end
end
def wat_helper(n)
@n = n
@a = "lol"
end
f = Foo.new(1)
p f #
f.do_wat(2)
p f #
wat_helper() isn't a member function of any class, so where do its member variables @n and @a bind? Whatever instance context it's called from, apparently.
This is powerful and terrible. On the one hand I could write some nice utility functions for maintaining a tree or a doubly-linked-list and inside any instance maintain @prev, @next, @less, @greater, @parent, and so on. That's great for re-use of the data structure utility functions.
But it's also terrible because I now kinda need to not trust any function I call because it might clobber or pollute my instance variables. There's a notion out there that one should code-review any open source project you're thinking about importing to see if its code is up to your standards. Practically things tend to get in on reputation and basic demonstration of working right. But if code I'm importing can potentially clobber my data structures in this weird way I feel like I should be extra suspicious of it.
https://bolson.org/~bolson/2016/wat_scope.rb
class Foo
def initialize(a)
@a = a
end
def do_wat(m)
wat_helper(m)
end
end
def wat_helper(n)
@n = n
@a = "lol"
end
f = Foo.new(1)
p f #
f.do_wat(2)
p f #
wat_helper() isn't a member function of any class, so where do its member variables @n and @a bind? Whatever instance context it's called from, apparently.
This is powerful and terrible. On the one hand I could write some nice utility functions for maintaining a tree or a doubly-linked-list and inside any instance maintain @prev, @next, @less, @greater, @parent, and so on. That's great for re-use of the data structure utility functions.
But it's also terrible because I now kinda need to not trust any function I call because it might clobber or pollute my instance variables. There's a notion out there that one should code-review any open source project you're thinking about importing to see if its code is up to your standards. Practically things tend to get in on reputation and basic demonstration of working right. But if code I'm importing can potentially clobber my data structures in this weird way I feel like I should be extra suspicious of it.
https://bolson.org/~bolson/2016/wat_scope.rb
Subscribe to:
Posts (Atom)