As China's appetite for technological advancement through acquisition grows, will we see a global Chinese-based software vendor emerge? READ MORE

[Open Source ]
View original post | Add to del.icio.us | Share

Hello and greetings from DrupalCon 2008 in Szeged, Hungary!

We (Thierry Manfé, Scott Mattoon and myself) are having a great time manning our booth and talking about Drupal, MySQL and Open Source@Sun with the nice crowd of Drupal Users and Developers here. Sun is a gold sponsor of the event and we're giving a number of sessions as well.

Today I gave my first presentation about MySQL Backup and Security - Best practices - unfortunately I ran a tad bit out of time at the end... The slides have already been attached to the session page, so you can read up on the last few things I was going to talk about. Feel free to contact me, if you have further questions!

Tomorrow I'll be talking about High availability solutions for MySQL: An Overview and practical demo, which will also include a practical demonstration of a two-Node Linux Cluster, performed by Jakub Suchy. In the afternoon, I will also hold a BoF about bzr - The Bazaar source revision control system

I've also uploaded some pictures from the event (and some impressions from the city) on my gallery (more will follow later). Enjoy!

[MySQL OSS backup collaborating community conference drupal gallery linux pictures presentation slides travel ]
View original post | Add to del.icio.us | Share

Is piracy of paid support a problem in the open source software market? READ MORE

[Open Source ]
View original post | Add to del.icio.us | Share

IDEs like NetBeans has made life easier and more productive. 

But, you may as well use utilities like 'cscope' when all you have is a console login. Here is how you would set up MySQL sources for easy cross-referencing using 'cscope':

Build the cross-reference:

Go the MySQL sources top-level directory and build the cross-reference (-R is used to descend recursively into the directories, and -b builds the cross-references):

/home/as227057/dev-tools/mysql-5.1.25-rc" 24 $ cscope -Rb

Search for symbols, functions, etc:

$cscope` -R brings up a text based interface:

Find this C symbol: Find this global definition: Find functions called by this function: Find functions calling this function: Find this text string: Change this text string: Find this egrep pattern: Find this file: Find files #including this file: Find all function definitions: Find all symbol assignments:

Using the various options available you can navigate through the sources easily. Here are some links on 'cscope':

• http://cscope.sourceforge.net/
  
• http://cscope.sourceforge.net/large_projects.html

View original post | Add to del.icio.us | Share

I had posted some notes on the MySQL test framework in my earlier post at http://blogs.sun.com/amitsaha/entry/the_mysql_test_framework

'mysql_client_test.c' which lives in the 'tests/' directory uses the MySQL C API to write various client side tests to be executed on the server. In this post, I will show how you can add your own test(s) to this file.

Why would you want to do that?

Adding your tests to this file enables you take advantage of the existing test framework to run your tests.

How does 'mysql_client_test.c' execute the tests?

In the main( ) function, this code snippet selects the tests to be run:

for ( ; *argv ; argv++) 18054 { 18055 for (fptr= my_tests; fptr->name; fptr++) 18056 { 18057 if (!strcmp(fptr->name, *argv)) 18058 { 18059 (*fptr->function)(); 18060 break; 18061 } 18062 } 18063 if (!fptr->name) 18064 { 18065 fprintf(stderr, "\n\nGiven test not found: '%s'\n", *argv); 18066 fprintf(stderr, "See legal test names with %s -T\n\nAborting!\n", 18067 my_progname); 18068 client_disconnect(); 18069 free_defaults(defaults_argv); 18070 exit(1); 18071 } where 'fptr' is defined as: struct my_tests_st *fptr

 

Map-like data structure

'my_tests_st' implements a map-like data structure which 'maps' test names to the function pointers which points to functions which contain codes for carrying out the tests. It is defined as:

struct my_tests_st { const char *name; void (*function)(); }; For eg. static struct my_tests_st my_tests[]= { 17681 {"testse_create_table", testse_create_table }, /* Entry for the new test method introduced - Amit.Saha@sun.com*/ 17682 { "disable_general_log", disable_general_log }, . .

'mysql_client_test.c' can operate in 2 modes: 

1. Run all the tests which are defined in the file (when no arguments is specified)
   
2. Run only the test which is specified in the argument list supplied to it. When the test is specified by its name, the structure above is looked up to find the appropriate function pointer and subsequently the function is called
   

Adding a new test case:

1. Write your test case in a new function and append it to 'mysql_client_test.c'
   
2. Append the method signature to the file somewhere towards the beginning of the file (after the #include statements)
3. Add the appropriate mapping in the static struct my_tests_st my_tests[] either at the end or beginning
4. Do a 'make' in the tests/ sub-directory
5. To run only the new test case, you just added: $./mysql_client_test function_name
   

Demo

Say, I want to create a new test case which will test whether I can create a table in my new custom storage engine- TESTSE.

So I will append the function to 'mysql_client_test.c':

/* Test to create a table using the TestSE */ void testse_create_table() { myheader("testse_create_table"); char QUERY[]="CREATE TABLE demo2(car CHAR(15) NOT NULL, year INTEGER NOT NULL) ENGINE=TESTSE"; printf("\n Client connected to MySQL Server %s \n", mysql_get_server_info(mysql)); printf("\n Connection Description %s\n", mysql_get_host_info(mysql)); if(mysql_query(mysql, QUERY)!=0){ printf("\n Query %s failed", QUERY); } else { printf("Query succeeded\n"); } }

Now add the following to static struct my_tests_st my_tests

{"testse_create_table", testse_create_table },

• Add a function signature for your function- void testse_create_table() somewhere before the place where the above mapping occurs
• Do a 'make'
• Now execute your test by doing:

$ ./mysql_client_test --no-defaults --socket=/tmp/mysql_9090.sock --user=root testse_create_table

Based on your system settings you may have to supply different options and values. The various options available can be seen by:

$ ./mysql_client_test --help

The test output should be something like:

##################################### client_connect ##################################### Establishing a connection to '' ...OK Connected to MySQL server version: 5.1.24-rc (50124) Creating a test database 'client_test_db' ...OK ##################################### 1 of (1/1): testse_create_table ##################################### Client connected to MySQL Server 5.1.24-rc Connection Description Localhost via UNIX socket Query succeeded ##################################### client_disconnect ##################################### dropping the test database 'client_test_db' ...OK closing the connection ...OK All '1' tests were successful (in '1' iterations) Total execution time: 0 SECS

View original post | Add to del.icio.us | Share

Windows FAIL.

It has been suggested the current thing I’m trying to fix is actually a bug in the Microsoft linker…. and I’m quite willing to believe that.

I wonder if I can expense rehab if this Windows port leads to a drinking problem….

[mysql ]
View original post | Add to del.icio.us | Share

Properties: 

Applicable To	InnoDB
Server Startup Option	--innodb_log_file_size=<value>
Scope	Global
Dynamic	Yes
Possible Values	Integer: Range: 1M - 4G <1M will be adjusted to 1M
Default Value	5M
Category	Performance, Maintenance

Description:

This variable defines the size of each log file in a log group. While setting this variable it should be noted that combined size of all log files should be less than 4GB.

InnoDB requires these logs for recovery in case of a crash. So how come the size of these logs effect server performance? As stated in MySQL manual "The larger the value, the less checkpoint flush activity is needed in the buffer pool, saving disk I/O.", these logs help InnoDB in running more confidently as it knows that even if data is not written to the persistent storage often it can still have it.

Best Value:

A larger value helps you in performance but only up to some point. After a certain value, the performance gain will be minimal or it can be negative. Another issue to be considered is that a value too large will slow down recovery as there will be more and more logs to be scanned. But definitely the default is too small.

My usual recommendation is to set it to 256M or if you feel its big (because maybe you have too many crashes and of course crash recoveries) then 128M. Anything beyond this range should be tested properly and justified.

How to set?

If you just change the size of this variable, MySQL will crib about the changed log file size and start without the InnoDB engine. The safe way of re-setting this value is:

1. Stop the MySQL server
2. Backup your data and log files
3. Delete log files
4. Set the new value for innodb_log_file_size in my.cnf
5. Start mysql server

Read More:

• MySQL manual entry on innodb_log_file_size 
• Choosing proper innodb_log_file_size @mysqlperformanceblog.com 
• Eric Bergen on innodb_log_file_size 
• Increase innodb_log_file_size: the proper way

View original post | Add to del.icio.us | Share

I have a small EC2 instance running with a 25GB EBS volume attached. It has a database on it that I need to manipulate by doing things like dropping indexes and creating new ones. This is on rather large (multi-GB, millions of rows) tables. After running one DROP INDEX operation that ran all day without finishing, I killed it and tried to see what was going on. Here’s the results of the first 10 minutes of testing:

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 0.818328 seconds, 160 MB/s

This looks great. I’d love to get 160MB/s all the time. But wait! There’s more!

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=100000 dd: writing `/vol/128.txt': No space left on device 86729+0 records in 86728+0 records out 11367641088 bytes (11 GB) copied, 268.191 seconds, 42.4 MB/s

Ok, well… that’s completely miserable. Let’s try something in between.

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 15.4684 seconds, 84.7 MB/s

So the performance gets cut in half when the number of 128k blocks is increased 10x. This kinda sucks. I’ll keep plugging along, but if anyone has hints or clues, let me know. If this is the way it’s going to be, then this is no place to run a production, IO-intensive (100,000s and maybe millions of writes per day, on top of reads) database.

<script type="text/javascript"> addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F08%2F27%2Fsustained-io-on-ebs-no-bueno%2F'; addthis_title = 'Sustained+IO+on+EBS+%3D%3D+No+Bueno'; addthis_pub = 'jonesy'; </script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12"></script>
[Database Linux Sysadmin Technology Web Services ]
View original post | Add to del.icio.us | Share

Code and details are here.

Use it for performance debugging rather rather than for performance. It may not make the server run faster, but it can show you where there are problems in the server code and you can fix those problems or report them to others.

The patch is for MySQL 5.1.26 and builds on x86_64 servers running Linux 2.6. It might not build elsewhere.

View original post | Add to del.icio.us | Share

Or, “Missing information in the MySQL Manual”.

Just earlier today, I was using POW(), which I’ve grown quite fond of, simply because it makes life easier. I prefer using it like SELECT 512*POW(1024,2) to find out the number of bytes to put in a variable, for example.

First, let’s take a look at the POW function:

Name: 'POW' Description: Syntax: POW(X,Y) Returns the value of X raised to the power of Y.

Okay, so it gives us a value; but what about the data type? Let’s take 512*POW(1024,2) as an example.

5067 (blogs) > SELECT 512*POW(1024,2) AS example; +-----------+ | example | +-----------+ | 536870912 | +-----------+ 1 row in set (0.00 sec)

What is that? Well, it sure does look like an INT at this point, doesn’t it?

(more…)

[MySQL abs data type mathematical functions pow ]
View original post | Add to del.icio.us | Share

Having a status column is very common in databases today. It can be used to denote a user status: CREATE TABLE IF NOT EXISTS `user` ( `user_id` int(10) unsigned NOT NULL auto_increment, `email` varchar(32) NOT NULL, `pw_hash` char(40) NOT NULL COLLATE latin1_general_cs, `status` enum('PENDING', 'ACTIVE', 'DISABLED') default 'PENDING', `date_created` timestamp NOT NULL default CURRENT_TIMESTAMP, PRIMARY KEY (`user_id`), UNIQUE KEY `idx_email` (`email`) ); or user-uploaded [...]
[MySQL MySQL Performance MySQL Schema Design ]
View original post | Add to del.icio.us | Share

Although the awareness of open source appears low, the lack of Internet-based apps poses an opportunity to bolster open source penetration READ MORE

[Open Source ]
View original post | Add to del.icio.us | Share

I was trying to check out the new Zabbix beta's when I ran into a couple of problems. Mostly missing dependencies but also some bugs.

The big troubles start when trying to install the frontent.
It seems like php code shipped in the last release might well work when you are upgrading but it fails when you are doing a fresh install. As I never got a Setup Wizard.

The clue is to get the most recent code from their SVN repo svn co svn://svn.zabbix.com/trunk , that actually allows you to run the setup wizard.

If you are installing the latest RPM's they are also not really requiring all the dependencies you need to run the frontend.

yum/apt-get install php-bcmath php-pear

Will get you a step further.

Another thing is the new password policy.. it seems the Admin/blank is gone.
I have no clue what the new password is but you are encouraged to set it yourselve in the mysql database

md5('new_pass') or echo -n new_pass | md5sum will help you .

Big thnx to the friendly folks over at #zabbix on irc.freenode.net for helping me out earlier.

Next steps .. debugging autodiscovery :)

[opensource zabbix ]
View original post | Add to del.icio.us | Share

A lot of people have been waiting for this a long time - so I am very happy to announce that we will publish the first Alpha that runs natively on Linux end of September. We will release generic binaries and the 5.1 source tarball including everything that is needed to build yourself.

This will not be a feature-complete beta release yet, but we are trying to get the basic functionality working so everybody can start modeling right away. We spent most of the last few months working on updating the Back-End implementation to use the GRTv4 and to prepare our platform-independent mForms. I expect the actual Front-End port to make good progress.

Let me share two screenshots I just made on Ubuntu, our Linux distro of choice. The first one shows the MySQL Model page. The Sakila demo model has been loaded.

The second screenshot shows the EER Diagram. This first Alpha will still use the software rendering option but we are looking forward to enable the OpenGL pretty soon.

Keep monitoring the blog - we will publish new screenshots as we go.

[Announcements ]
View original post | Add to del.icio.us | Share

(Author's note: not necessarily actually a practical idea. But fun!)So pictured here is a histogram of a moderately large set of random integers. Each vertical line represents the total number of entries at each particular integer. Since each number is made up of multiple random factors (10 different random numbers, each between 0 and 100, added together), the distribution tends toward a bell curve.So how did I build the graph? Excel? PHP? Nope. Just a MySQL query. ...

View original post | Add to del.icio.us | Share

Short Version: You can find a fantastic video here about bundling customized AMIs and registering them with Amazon so that you can launch as many instances of your new AMI as you want. The video is so good that I don’t bother writing out the steps to do the bundling (it would be pretty darn long). These are some notes about launching an AMI, customizing it, and mounting an EBS volume to it (the video linked above doesn’t cover EBS). Also, check out the ElasticFox tool which is a very good GUI for doing simple EC2 operations. Nice if you’re just getting started or doing some simple tests.

There are two ways you can go about creating a custom machine image (AMI) for use with Amazon EC2: You can create an image locally by dd’ing to a file, mounting it with “-o loop” creating a filesystem on it, and bootstrapping the whole thing yourself, or you can grab an existing AMI that will serve as a “good enough” base for you to make your customizations, then bundle the customized image.

I’ll be talking about the latter option, where you identify a “good enough” image, customize it for your needs, and save that as your AMI. Unless you’re doing some kind of highly specialized installation, or are a control freak, you shouldn’t really need to start from scratch. I was just building a test image, and wanted a CentOS 5.2 base installation.

Here’s the command you can use to browse the AMIs you have access to (they’re either public, or they’re yours):

$ ec2dim -a

If that command looks funny to you, it’s likely because you’re used to seeing the really long versions of the AWS commands. Amazon also provides shorter versions of the commands. No, really - have a look! The long version of this command is:

$ ec2-describe-images -a

Too long for my taste, but it’s nice to know it’s there.

So, rather than start from scratch, I grabbed a base image that was close enough for my needs, and customized it. It’s a 5.1 base image, pretty well stripped of things that I don’t need, and a few that I do, but that’s ok. I’d rather start with less than more.

So step one is to launch an instance of the AMI I’ve chosen to be my ‘base’. Simple enough to do:

$ ec2run ami-0459bc6d -k ec2-keypair

And that’s pretty much it. It takes a couple of minutes (literally) for the machine to actually become available. You can check to see if it’s still in “pending” state or if it’s available by running ‘ec2din’. Without arguments, that’ll show you the status of any instances you have pending or runnning. Once the instance is running, you’ll be able to glean the hostname from the information provided.

An important note at this point: Don’t confuse “image” with “instance”. For the OO types in the crowd, an “image” is an object. It does nothing by itself until you instantiate it and create an “instance” of that object. For sysadmins, the “image” is like a PXE boot image, which does nothing until you boot it, thereby creating an “instance”.

The reason I used “PXE” and “object” in the above is because of the implication it makes: you can launch as many instances of an object as you want from a single object definition. You can boot as many machines as you want from a single PXE boot image. Likewise, you can launch as many Amazon EC2 instances from an image as you want.

So, in the time it took you to read those last two paragraphs, your instance is probably running. I now grab the hostname for my instance, and ssh to it using my keypair:

$ ssh -i ec2-keypair root@<hostname>

Now that I’m in, I can customize the environment, and then “bundle” it, which will create a new AMI with all of my customizations. With the instance in question, I installed a LAMP stack, and a few other sundry tools I need to perform my testing. I also ran “yum -y upgrade” which will go off and upgrade the machine to CentOS 5.2.

One thing I want to do with this instance is test out the process for creating an EBS volume. The two pieces of information I need to do this are the size of the volume I want to create, and the “zone” I want to create it in. You can figure out which zone your instance is running in using ‘ec2din’ on your workstation (not in your instance). I took that information and created my image in the same zone using the ‘ec2addvol’ command. If you don’t have that command on your workstation, then you don’t have the latest version of the Amazon command line tools. Here’s the command I ran:

$ ec2addvol -z us-east-1b -s 25

To see how it went, run ‘ec2dvol’ by itself and it’ll show you the status of all of your volumes, as well as the unique name assigned to your volume, which you’ll need in order to attach the volume to your instance. To do the ‘attachment’, you need the name of the volume, the name of the instance (use ‘ec2din’), and you need to choose a device that you’ll tell your instance to mount. Here’s what I ran (on my workstation):

$ ec2attvol -d /dev/sdx -i i-xxxxxxxx -v vol-xxxxxxxx

Now you can go back to the shell on your instance, mount the device, create a file system, create a mount point, add it to fstab, and, as they say in the UK, “Bob’s yer uncle”. By the time I wrote this post, I had already shut down my instance, but here are the commands (caveat emptor: this is from memory):

# mkfs.ext3 /dev/sdx # mkdir /vol # mount /dev/sdx /vol

If that all works ok, you can add a line to /etc/fstab so that it’ll be mounted at boot time, but I haven’t yet figured out how to attach a volume to an instance at boot time. The mount doesn’t work if you don’t attach the volume to the instance first. You’ll get a “device doesn’t exist” error if you try it. Clues hereby solicited. I assume I could probably use ‘boto’ and some Python code to get this done, but doing the same with a shell script wrapper around the Amazon tools might also be possible — but I don’t know how reliable that would be, because you’re at the mercy of Amazon and how they decide their tools should present the data (and *if* they provide the data you need for a particular operation down the road).

So now I have an EBS volume, and an instance. The volume is attached to the instance, and I can do things with it. I’m testing some database stuff, so I copied a database over to the volume, which was now mounted, so I could just ’scp mydb.tbz root@<instance>:/vol/.’

Once my database is there, I can attach it to pretty much whatever I want, which makes it nice, because I can test the same database, and the same database code, and see how the different size Amazon instances affect the performance, which gives me more performance data to work with. For production purposes, I’ll have to look more closely at the IO metrics, play with attaching multiple volumes and spreading out the IO, and I also want to test the ’snapshot’ capabilities. It’s also nice to know that if I needed to launch this in production (there are no plans to do so, but you never know), I could upgrade the database “hardware” more or less instantly

If anyone has code or tools to help automate the management of all of this stuff, please send links! If I come up with any myself, I’ll most likely post it here.

Now that I have a customized AMI with all of my packages installed and my config changes made, I need to bundle this so that I can boot as many instances of this particular configuration as I want. An important note about bundling this *particular* image is that you MUST run ‘depmod -a; modprobe loop’ before bundling, since this process basically abstracts the manual process of bundling an image, which involves mounting a file as a volume, which requires a loopback mount.

The video I used to do the bundling is here, and if you can live through the disgustingly bad burps and chirps in the (Flash version) audio, it’s an excellent tutorial for bundling custom AMIs. While the process *is* pretty straightforward, it involves a number of steps, and the video goes through all of them, and it worked perfectly the first time through.

<script type="text/javascript"> addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F08%2F27%2Fmore-adventures-in-amazon-ec2-and-ebs%2F'; addthis_title = 'More+Adventures+in+Amazon+EC2+%28and+EBS%29'; addthis_pub = 'jonesy'; </script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12"></script>
[Database Python Sysadmin Technology Web Services ]
View original post | Add to del.icio.us | Share

I was surprised to find yesterday when using MySQL 5.1.26-rc with a client I’m recommending 5.1 to, some information not seen in the EXPLAIN plan before while reviewing SQL Statements.

Using join buffer

+----+-------------+-------+--------+---------------+--------------+---------+------------------------+-------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+--------------+---------+------------------------+-------+----------------------------------------------+ | 1 | SIMPLE | lr | ALL | NULL | NULL | NULL | NULL | 1084 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | ca | ref | update_check | update_check | 4 | XXXXXXXXXXXXXXXXX | 4 | Using where; Using index | | 1 | SIMPLE | ce | ALL | NULL | NULL | NULL | NULL | 13319 | Using where; Using join buffer | | 1 | SIMPLE | co | eq_ref | PRIMARY | PRIMARY | 4 | XXXXXXXXXXXXXXXXX | 1 | Using where | +—-+————-+——-+——–+—————+————–+———+————————+——-+———————————————-+ 4 rows in set (0.00 sec) mysql> select version(); +———–+ | version() | +———–+ | 5.1.26-rc | +———–+ 1 row in set (0.00 sec)

Sergey Petrunia of the MySQL Optimizer team writes about this in Use of join buffer is now visible in EXPLAIN.

View original post | Add to del.icio.us | Share

I came across the OStatic Free hosting service that provide Solaris + Glassfish (Java Container) + MySQL.

They offer “Now you can get free Web hosting on Cloud Computing environment free of charge for up to 12 months. ”

The catch “accumulate 400 Points for participating on the site“.

A rather novel approach, you get 100 points for registering, but then only 5-15 points per task. The equates to approximately at least 30 tasks you need to perform, such as answering a question, or reviewing somebodies application. That seems like a lot of work?

In this case, Free has definitely a cost and time factor to consider.

View original post | Add to del.icio.us | Share

Hi all,

The schedule for PyWorks has been posted! I’m really excited about three things:

1) there are some really cool talks that I’m looking forward to attending. There are a couple of sysadmin-related talks, AppEngine, TurboGears, Django, and an area I’ve been especially slow to move into: testing (I know, shame on me). There’s lots more so be sure to check it out.

2) the conference scheduling process is over

3) I get to meet a lot of people face-to-face that I’ve worked with in the past on Python Magazine developing articles, or interacted with on IRC, etc. One thing I like about conferences surrounding open source technologies is you get to thank people face-to-face for the sweat they poured into some of the tools I use regularly. Mark Ramm, Kevin Dangoor, Michael Foord, Brandon Rhodes, and a collection of Python Magazine authors will be speaking there, and other Python Magazine folks and generally familiar faces will be in attendance.

Enjoy!

For those still unaware, PyWorks will be held in Atlanta, Nov. 12-14, 2008. It’s sponsored by MTA, the publisher of Python Magazine, as well as php|architect. In fact, the php|works conference will be held simultaneously with PyWorks, and attendees of one are free to access talks in the other at will. There will also be a “center track” that will cover some more generic topics of interest to developers without regard to the language in use. Check it out!

<script type="text/javascript"> addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F08%2F27%2Fpyworks-conference-schedule-posted%2F'; addthis_title = 'PyWorks+Conference+Schedule+Posted'; addthis_pub = 'jonesy'; </script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12"></script>
[Database PHP Python Scripting Sysadmin Technology Web Services ]
View original post | Add to del.icio.us | Share

James Kobelius writes in Network World how the need for scalable real-time business intelligence will create a convergence of technologies centered on database virtualization:
"Real-time is the most exciting new frontier in business intelligence, and virtualization will facilitate low-latency analytics more powerfully than traditional approaches. Database virtualization will enable real-time business intelligence through a policy-driven, latency-agile, distributed-caching memory grid that permeates an infrastructure at all levels.

As this new approach takes hold, it will provide a convergence architecture for diverse approaches to real-time business intelligence, such as trickle-feed extract transform load (ETL), changed-data capture (CDC), event-stream processing and data federation. Traditionally deployed as stovepipe infrastructures, these approaches will become alternative integration patterns in a virtualized information fabric for real-time business intelligence." Kobelius makes it clear that this "virtualized information fabric" is an ambitious program that will be accomplished only over a number of years, but the underlying trends are visible now: for example, the convergence of distributed caches with databases, as evidenced by Oracle's acquisition of Tangosol, and Microsoft's recently announced Project Velocity.

This envisioned system contains so many moving parts that a new paradigm is will be needed to link them together. I don't think that databases are the answer. They elegantly handle stored data, but founder when dealing with change, caching, and the kind of replication problems you encounter when implementing virtualized and distributed systems. For example, database triggers are the standard way of managing change in a database, and are still clunky fifteen years after they were introduced; and Enterprise Information Integration (EII) systems were an attempt to extend the database model to handle federated data, but only work well for a proscribed set of distribution patterns.

I wrote recently about how SQLstream can implement trickle-feed ETL and use the knowledge it gleans from the passing data to proactively manage the mondrian OLAP engine's cache. SQLstream also has adapters to implement change-data capture (CDC) and to manage data federation.

In SQLstream, the lingua franca for all of these integration patterns is SQL, whereas ironically, if you tried to achieve these things in Oracle or Microsoft SQL Server, you would end up writing procedural code: PL/SQL or Transact SQL. Therefore streaming SQL - a variant of what Kobelius calls event-stream processing where, crucially, the language for event-processing language is SQL - seems the best candidate for that unifying paradigm.

View original post | Add to del.icio.us | Share

OpenSQL Camp 2008 is coming! When and Where: Charlottesville, Virginia, USA November 14, 15, and 16 2008!

Organised by Baron Schwartz & others, and attended by loads of cool and interesting people (Brian, Monty and Baron are already on the attendee list!) you'd better get ready for a dynamite weekend of learning, contributing, and having fun! I'm hoping to be there too.

Some Key facts:

• It is of, by and for the community (you).
  
• At this event, all Open Source databases are created equal. We?ll learn together and grow together.
  
• It?s a combination conference and hackathon.
  
• It?s free.
  
• It is Friday night Nov 14, 18:00 through Sun the 16th at 18:00 in Charlottesville, Virginia USA in a very cool location.
  
• The website, where all details will be posted: http://opensqlcamp.pbwiki.com/
  
• The mailing group, where details will be discussed, decided and arranged: http://groups.google.com/group/opensqlcamp
  
• Date, time and place are confirmed and will not change. You can make your travel plans now. There?s travel information on the wiki.
  
• We already have some great speakers who have offered to give great talks. These and other details will show up on the wiki as they?re finalised.
  
• Sponsors are needed. See the wiki.

Despite the name, this will be different from other Camp conferences you?ve been to. This is a combination of a planned event (with great speakers and sessions), semi-planned spontaneity (sessions to be decided by attendees the night before), and a hackfest. It?s the best elements cherry-picked from all the conferences (and un-conferences) you?ve been to.

What should you do next? Go to the website and register yourself. Then go join the mailing list. And buy your plane tickets before they get too expensive. And tell your friends, and blog about it. See you there!

View original post | Add to del.icio.us | Share

Aka, colliding MD5, but in a very cool 12-way demonstration:
We have used a Sony Playstation 3 to correctly predict the outcome of the 2008 US presidential elections. In order not to influence the voters we keep our prediction secret, but commit to it by publishing its cryptographic hash on this website. The document with the correct prediction and matching hash will be revealed after the elections.Read all about it at http://www.win.tue.nl/hashclash/Nostradamus/

(yes I share my first name with one of the authors, but you'll notice that the last name, while similar, is not identical - it's really not me. I'm not that much of a maths or crypto whiz ;-)

View original post | Add to del.icio.us | Share

A couple of article have been published recently that point to a growing realisation/admission about the role that open source will play in the future of enterprise software.

In “The Commercial Bear Hug of Open Source” Dan Woods details the various methods by which open source has become increasingly commercial in recent years, while in “The Microsoft-Novell Deal and Trust in Princes” Bruce Byfield discusses the relationship between business and open source.

Neither article is perfect. Woods, in particular, appears to paint open source in the role of the glorious failure - failing to surpass traditional licensing models and being subsumed into the mainstream (a subject I’ve touched on before).

For his efforts Woods earns the wrath of Dana Blankenhorn, who points out that Woods has confused the idealism of free software and Richard Stallman with the pragmatism of the open source and Eric Raymond.

“It [Raymond's 'open source' concept] accepted the idea of commercial interests from the start,” writes Blankenhorn. “It saw new business models evolving from shared development effort.”

For that reason, Woods asking “Isn’t open source a community-based movement that was set to overtake the world of commercial software? Wasn’t the famous LAMP stack, Linux, Apache, MySQL and Perl, Python and PHP going to open a world in which software existed outside the traditional realm of property?” is something of a straw man.

However, his central point that “commercial and open source are fellow travelers” is valid, as his his point that “it is almost impossible to tell the difference between the most popular open source software and commercial alternatives”.

That brings us to Byfield’s article in Datamation. While it is ostensibly a discussion about the extension of Microsoft’s Linux deal with Novell, the article focuses heavily on the relationship between business and free and open source software (FOSS).

“What tends to get lost is this: the fact that business is friendly to FOSS does not mean that it has adopted its values,” he writes. “The free software camp’s concern with philosophical and political freedom has almost certainly not been adopted by most FOSS-friendly companies, while the open source camp’s emphasis on increased software quality is probably shared by middle-management at best. Business –gasp!– is interested in FOSS to improve the bottom line, and often no other reason.”

This is a statement that leaves Matt Asay with mixed emotions. “Commercial open source is starting to find its feet,” he writes. “There isn’t a grand contradiction between giving code away (seeding the market) and suggesting a purchase (reaping the market). In fact, the two go together perfectly. As we realize this, open source will become even more dominant. Open source and business can be bosom buddies, not enemies.”

I would maintain that the relationship is closer than that. However, from Byfield’s viewpoint there remain barriers. “Sooner or later, an open source business is going to act more like a business and less like a citizen of the FOSS community,” he warns. “The two can certainly co-exist, and both can benefit from doing so. But, forced to choose, the average FOSS-based business is going to choose business interests over FOSS every time,” he adds.

The tone of Byfield’s article is driven in part by his evident belief that business and open source are at least partially exclusive. This, like Woods’ confusing of free and open source software, is a mistake in my opinion.

It is worth remembering that open source is a business tactic, not a business model. Open source is not a market in and of itself, nor is it a vertical segment of the market. Open source is a software development and/or distribution model that is enabled by a licensing tactic. It enables new revenue generation strategies.

Companies that build revenue streams around open source software do not choose between business and open source, they choose business and open source. You could call this a commercial bear hug of open source, or you could see it as an evolution of commercial business models based on the economic principle that sharing development has the capacity to lower development costs.

Either way the result is the same - the increased adoption of open source as a development and distribution model by mainstream business. Which is, after all, what the open source movement is all about.

See also:
Commercial licensing is a double-edged sword
Asking the right questions of open source
Open source: assimilate and thrive
Judging open source business models

[Software 451 group 451caostheory 451group bruce byfield business caostheory dan woods Dana Blankenhorn Linux matt aslett matta asay mattaslett matthew aslett matthewaslett open-source opensource The 451 Group the451group ]
View original post | Add to del.icio.us | Share

You will need:

• CMake (at least 2.4.7)
• Bazaar (the newer the better - 1.6 was just released - at least use that)
• Gnu Bison
• Visual Studio (Express works, but I’m talking about 2005 here)
• … and all this installed on a Microsoft Windows machine.
• … and to hate yourself, you are going to be using Windows after all.

Then, get and build it:

1. Get the source:
   bzr branch lp:~mysql/mysql-server/mysql-5.1-telco-6.4-win
2. Run CMake. the CMake GUI can now be used to select compile options! You’ll have to set the path “where is the source code” to where you put the source code in step 1.
3. Hit “Configure” in CMake
4. Select the target (i.e. the version of Visual Studio you’re going to use)
5. Select the build options. HINT: WITH_NDBCLUSTER_STORAGE_ENGINE may be a useful one to enable
6. Hit Configure again
7. Hit Ok.
8. CMAKE now generates the Visual Studio project. Use this time to drink some good scotch.
9. Open Mysql.sln (which should launch Visual Studio)
10. Go Build -> Build Solution (or hit F7)

Now you can go and have much whisky as this will take a few minutes. You should now have a set of built binaries for MySQL Cluster on Windows. Scary.

[mysql ]
View original post | Add to del.icio.us | Share

Along with some others, I have arranged a conference for open-source database users and developers.

Key facts:

• It is of, by and for the community (you).
• At this event, all open-source databases are created equal. We’ll learn together and grow together.
• It’s a combination conference and hackathon.
• It’s free.
• It is Friday night Nov 14, 18:00 through Sun the 16th at 18:00 in Charlottesville, Virginia USA in a very cool location.
• The website, where all details will be posted: http://opensqlcamp.pbwiki.com/
• The mailing group, where details will be discussed, decided and arranged: http://groups.google.com/group/opensqlcamp
• Date, time and place are confirmed and will not change. You can make your travel plans now. There’s travel information on the wiki.
• We already have some great speakers who have offered to give great talks. These and other details will show up on the wiki as they’re finalized.
• Sponsors are needed. See the wiki.

Despite the name, this will be different from other Camp conferences you’ve been to. This is a combination of a planned event (with great speakers and sessions), semi-planned spontaneity (sessions to be decided by attendees the night before), and a hackfest. It’s the best elements cherry-picked from all the conferences (and un-conferences) you’ve been to.

What should you do next? Go to the website and register yourself. Then go join the mailing list. And buy your plane tickets before they get too expensive. And tell your friends, and blog about it.

See you on the mailing list and the wiki!

conference
[OpenSQL Camp SQL conference ]
View original post | Add to del.icio.us | Share