After reading few books, digging Internet, finally I found way to use mootools Ajax in Joomla!

Point is in format=raw. If you have any component, if you add format=raw in link, for example

index.php?option=com_ajax&Itemid=60&format=raw you will see only result of component, and that what you need for ajax result.

You can use combination of "standard" ajax and Joomla! but in that case you would loose some of functionality of Joomla!, so better way is to use straight Joomla!

Here is my example, which you can download from here:

 

<?php  // no direct access
  defined( '_JEXEC' ) or die( 'Restricted access' );
  // add mootools
  JHTML::_('behavior.mootools');
  // Check the task parameter and execute appropriate function
  switch( JRequest::getVar( 'task' ))  {
    case 'result':
      searchResult();
      break;
    default:
      search();
    break;
  }
  function searchResult()
  {
    echo JRequest::getVar( 'name' );
  }
  function search()
  {
    $doc =& JFactory::getDocument();
    ?>
			<form method="post" action="<?php
						echo JRoute::_('index.php?option=com_regdavzav&task=result&format=raw'); ?>
					         " id="form1">  
  		  Ajax test: <input type="text" value="" class="flatinput" size="45" name="name" id="name"/>            
        <input type="submit" name="button132" id="button1" value="Submit"/>
		  </form>    
<div id="update"></div>
<?php
		$js = "window.addEvent('domready', function()
						{
							$('form1').addEvent('submit', function(e)
								{
									// Stop the form from submitting
									new Event(e).stop();
									// Update the page
									this.send({ update: $('update') });
								});
						});";
		$doc->addScriptDeclaration($js);
  }
?>

With JHTML::_('behavior.mootools'); you are including mootools framework in your component. $doc =& JFactory::getDocument(); you will use for adding javascript ($js = ...) And in <div id="update"></div> you are actually updating page with informations from Ajax. <div> must be in function search() otherwise you will have problems (to me happened that I always added result, instead of cleaning previous).

I hope that this is all.